Historic advances in computer technology have made it economical for individual users to have their own computing system, which caused the proliferation of the Personal Computer (PC). Continued advances of this computer technology have made these personal computers very powerful but also complex and difficult to manage. For this and other reasons, there is a desire in many workplace environments to separate the user interface devices, including the display and keyboard, from the application processing parts, or data processor of the computing system. In this preferred configuration, the user interface devices are physically located at the user's desktop, while the processing and storage components of the computer are placed in a central location. The user interface devices are then connected to the data processor and storage components with some method of communication.
Several commercial techniques exist to support the transmission of these user interface signals over standard networks and some are compared in “A Comparison of Thin-Client Computing Architectures,” Technical Report CUCS-022-00, Jason Nieh, S. Jae Yang and Naomi Novik, Network Computing Laboratory, Columbia University, November 2000.
While there are a variety of techniques used today to communicate video or graphics across a network of limited bandwidth, none utilize compression algorithms that take advantage of the recurring history patterns found within a computer display image.
One of the challenges associated with existing techniques lies in the development of methods capable of transmitting high bandwidth display signals from the processing components to the remote desktop across a standard network of relatively low bandwidth.
Some telecommunications technologies compress data streams based on recurring patterns; for example the compression of tones signals and other audio information. As one example, RFC2833 specifies a method for transmitting DTMF tones across a packet network using coded signatures rather than transmitting the tones themselves. Another example is the structured audio object language (SAOL) as found in MPEG-4, described in ISO/FDIS 14496-3, that is used to transmit complex audio across packet networks as a series of instructions. These methods support only a pre-defined range of input signals and generally expect the client side of the network to contain a static library of objects identified by the signature rather than dynamically detecting and using historic data patterns. Additionally, these methods are not applicable to video and digital graphics.
There are a few examples of methods for communicating computer display graphics information across a network. The simplest method is to periodically send copies of frame buffer information from a data processor. This is impractical for sending a normal SXGA display image of 1280×1024 at 24-bit color resolution across a standard 100 Base T LAN network as each 4 MB frame would take 0.3 seconds of dedicated network bandwidth, making perception-free communications of display information impossible.
An alternative approach is to intercept graphics instructions on the data processor and communicate these across the network to the remote user. This method is both restrictive in its compatibility and intrusive as it requires operating system dependent graphic command routing software running on the data processor. Furthermore, it requires a processor and software located at the remote user interface that is capable of interpreting the graphics commands. This adds cost and complexity to the remote installation.
Another approach is for the data processor to compare the previously transferred frame buffer information with the current frame and only transfer changes between them. This decreases the overall amount of data, especially for computer displays where much of the display is often static from frame to frame. This approach is expensive to implement as the data processor requires at least two frame buffers, a copy of the previously communicated frame buffer and the present frame. The previous frame buffer must be compared a pixel at a time with the present frame buffer and the difference communicated, either in real-time or from a temporary additional delta-buffer. This approach is memory- and computationally intensive, thus decreasing the performance of applications running on the data processor. This is especially noticeable in applications such as video that involve significant screen refresh activity where each screen refresh requires the movement and copying of graphics information between the frame buffers across the local system bus of the data processor.
An enhancement of this method reduces the overall data processor memory requirement by segmenting the frame buffer into spatially connected tiles and maintaining a list of signatures for the tiles. The new display frame is tiled and the signature for each new tile is compared with the signature in the list to determine if the tile should be transferred.
The concept of tiling images for remote display is popular and forms the basis for standard methods of remotely accessing images, including the JPIP protocol described in Part 9 of the JPEG2000 standard and the AT&T Laboratories RFB protocol. Variations on display tiling are described by Emerson et al. in U.S. Pat. No. 6,664,969 and Szamrej in U.S. Pat. No. 5,990,852. These tiling and list systems and methods are limited. They all require hardware or application-based frame buffers that are tightly-coupled with the data processor architecture. Any copying of display pixels or signatures loads the system bus of the data processor, lowering the performance of the system. Some of the methods interrupt the operating system for a background task to manage the activity, which severely reduces the performance of the data processor.
In summary, existing methods of transferring display information from a data processor to a remote user interface are limited by high cost and high maintenance requirements. The supported display update rates are low or the methods used adversely impact the performance of the data processor. These solutions only look for changes from one frame to the next. None of the systems described take advantage of the specific characteristics of a computer display, specifically the frequent transitions of parts of a display image to historic, previously-displayed states. None of these methods use a non-intrusive compression engine capable of perception-free, real-time compression of a digital raster signal.